store-gateway: receive bucket index metadata from queriers #13942
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
narqo
force-pushed
the
vldmr/store-gw-recv-bucket-index-version
branch
2 times, most recently
from
f7f4e2f to
8c1b022
Compare
Signed-off-by: Vladimir Varankin <[email protected]>
Signed-off-by: Vladimir Varankin <[email protected]>
Signed-off-by: Vladimir Varankin <[email protected]>
Signed-off-by: Vladimir Varankin <[email protected]>
Signed-off-by: Vladimir Varankin <[email protected]>
narqo
force-pushed
the
vldmr/store-gw-recv-bucket-index-version
branch
from
688ca2c to
db59721
Compare
narqo
changed the title
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
alexweav
approved these changes
Jan 8, 2026
pkg/storegateway/bucket.go
Outdated
| if diff > 0 { | ||
| group = labelDiscoveryDiffNewer | ||
| } else { | ||
| group = labelDiscoveryDiffOlder |
Contributor
There was a problem hiding this comment.
Our hypothesis that (typically) there should be zero cases, where queriers discover a new version of bucket-index faster than store-gateways
If this is the case, it might be cheaper to just log at warning level when it's unexpectedly older, instead of introducing new metrics that are always recorded but in theory will be entirely in the Newer category?
Contributor
Author
There was a problem hiding this comment.
Now as you mentioned this, I wonder if we really need this metric at all — how about we start with only a warning (and a debug for normal case)? WDYT? Ref 31d7e2d
Signed-off-by: Vladimir Varankin <[email protected]>
Signed-off-by: Vladimir Varankin <[email protected]>
narqo
force-pushed
the
vldmr/store-gw-recv-bucket-index-version
branch
from
0616302 to
0db1823
Compare
4 tasks
narqo
added a commit
that referenced
this pull request
Jan 16, 2026
#### What this PR does Following #13942 The PR fixes the bucket index discovery logging. In the previous one, I've missed that the fields added to the `With` are only applied to the fallback logger. So we don't see these fields in practice 🤦 I've noticed this only now because in one of the cells, where the changes were deployed we do see quite a fair bit of these warnings: <img width="1462" height="225" alt="Screenshot 2026-01-15 at 18 05 53" src="https://github.com/user-attachments/assets/f84144d4-0f67-4b06-a6b0-b15c23ce3ea0" /> #### Which issue(s) this PR fixes or relates to Relates to grafana/mimir-squad#3373 #### Checklist - [ ] Tests updated. - [ ] Documentation added. - [ ] `CHANGELOG.md` updated - the order of entries should be `[CHANGE]`, `[FEATURE]`, `[ENHANCEMENT]`, `[BUGFIX]`. If changelog entry is not needed, please add the `changelog-not-needed` label to the PR. - [ ] [`about-versioning.md`](https://github.com/grafana/mimir/blob/main/docs/sources/mimir/configure/about-versioning.md) updated with experimental features. Signed-off-by: Vladimir Varankin <[email protected]>
mimir-github-bot bot
pushed a commit
that referenced
this pull request
Jan 16, 2026
#### What this PR does Following #13942 The PR fixes the bucket index discovery logging. In the previous one, I've missed that the fields added to the `With` are only applied to the fallback logger. So we don't see these fields in practice 🤦 I've noticed this only now because in one of the cells, where the changes were deployed we do see quite a fair bit of these warnings: <img width="1462" height="225" alt="Screenshot 2026-01-15 at 18 05 53" src="https://github.com/user-attachments/assets/f84144d4-0f67-4b06-a6b0-b15c23ce3ea0" /> #### Which issue(s) this PR fixes or relates to Relates to https://github.com/grafana/mimir-squad/issues/3373 #### Checklist - [ ] Tests updated. - [ ] Documentation added. - [ ] `CHANGELOG.md` updated - the order of entries should be `[CHANGE]`, `[FEATURE]`, `[ENHANCEMENT]`, `[BUGFIX]`. If changelog entry is not needed, please add the `changelog-not-needed` label to the PR. - [ ] [`about-versioning.md`](https://github.com/grafana/mimir/blob/main/docs/sources/mimir/configure/about-versioning.md) updated with experimental features. Signed-off-by: Vladimir Varankin <[email protected]> (cherry picked from commit b268f75)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does
Following #13875 and #13963
The PR adds a newcortex_bucket_store_bucket_index_discovery_difference_totalmetric, that tracks how often store-gateway receives requests from querier instances, whose bucket-index version is difference from the version, the store-gateway aware of.This is how the debug data looked from a live dev cell (I'm not a huge fan of git-style "ours" vs. "theirs", etc naming; but that's what I came with)
Our hypothesis that (typically) there should be zero cases, where queriers discover a new version of bucket-index faster than store-gateways. To start, the new metric allows us to proof this theory.
I'm thinking that in the future, we may move the passed bucket index from gRPC context to a request's hints to make this (an optional) part of the API contract. The theory is that we can leverage the fact that querier and store-gateway are in agreement about their view a bucket-index (this is still handwave'y for the moment; no concrete details here).
Which issue(s) this PR fixes or relates to
Relates to https://github.com/grafana/mimir-squad/issues/3373